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ABSTRACT 

Experimentally comparing the energy usage and comfort 
characteristics of different controllers in heating, ventila- 
tion, and air-conditioning (HVAC) systems is difficult be- 
cause variations in weather and occupancy conditions pre- 
clude the possibility of establishing equivalent experimental 
conditions across the order of hours, days, and weeks. This 
paper is concerned with defining quantitative metrics of en- 
ergy usage and occupant comfort, which can be computed 
and compared in a rigorous manner that is capable of de- 
termining whether differences between controllers are sta- 
tistically significant in the presence of such environmental 
fluctuations. Experimental case studies are presented that 
compare two alternative controllers (a schedule controller 
and a hybrid system learning-based model predictive con- 
troller) to the default controller in a building-wide HVAC 
system. Lastly, we discuss how our proposed methodology 
may also be able to quantify the efficiency of other building 
automation systems. 

Categories and Subject Descriptors 

1.2.8 [Artificial Intelligence]: Problem Solving, Control 
Methods, and Search — control theory, performance, mea- 
surement; J. 7 [Computer Applications]: Computers in 
Other Systems — command and control 

General Terms 

Experimentation, Measurement, Performance 

Keywords 

HVAC, energy-efficiency, comfort, metrics, control 

*Open source software that implements our methodology in 
the MATLAB language can be found at the following link 
http : / /hybrid. eecs .berkeley.edu/~NEDE/ compEng . zip 



1. INTRODUCTION 

Heating, ventilation, and air-conditioning (HVAC) systems 
contribute to a significant fraction of building energy usage. 
As a result, these systems have seen an increasing amount of 
research towards their modeling and efficient control (e.g., 
[T6|[T0|[17|[13)[3)[14]). The primary challenge is ensuring 
comparable levels of occupant comfort, in relation to exist- 
ing HVAC controllers, while achieving reductions in energy 
consumption. Simulations and experiments indicate that 
this goal is achievable for a large variety of systems. 

Experimentally comparing the energy efficiency and comfort 
of different control schemes is difficult because of the large 
temporal variability in weather and occupancy conditions. 
Furthermore, energy usage of HVAC equipment does not 
typically scale linearly in the relevant variables. For exam- 
ple, a simplistic model would state that the amount of en- 
ergy E required to maintain a set point of T s for the supply 
air temperature (SAT) of an HVAC system with a warmer 
outside air temperature (OAT) of T (> T s ) is proportional 
to E ~ T — T s . We have observed on our building-wide 
testbed [5] that such models do not capture the complex en- 
ergy characteristics of the equipment. One reason for this is 
that some equipment is designed to operate most efficiently 
at certain temperatures or settings. 

There is an additional difficulty in experimentally comparing 
the efficiency of different control schemes. Not all buildings 
have equipment to directly measure the energy consumption 
of only HVAC equipment. It is common for energy mea- 
surements to include appliances, water heating, and other 
sources of energy consumption that are difficult to disaggre- 
gate from HVAC energy usage. Such disaggregation is hard 
because the energy signature and characteristics of HVAC 
equipment is often similar to appliances like refrigerators or 
water heaters. Another reason that disaggregation is dif- 
ficult is that installing separate meters to measure energy 
usage of an HVAC system that is physically and electrically 
distributed throughout a building can be cost-prohibitive. 
As a result, experimental comparison may require compar- 
ing measurements for which the absolute differences are a 
small percentage of the total. Determining whether such 
differences are statistically significant can be challenging. 



The purpose of this paper is to propose a set of quantita- 



Type 


Description 


Option A 


Measuring key variables that affect 
HVAC energy consumption 


Option B 


Directly meausuring HVAC energy 
consumption 


Option C 


Measuring whole building or sub- 
building energy consumption 


Option D 


Estimating HVAC energy consumption 
using simulation models 



Table 1: The International Performance Measure- 
ment and Verification Protocol (IPMVP) |6 cate- 
gorizes approaches to comparing energy usage into 
four groups. 

tive methods for comparing different HVAC control schemes. 
We define quantitative metrics of efficiency and comfort, 
and we introduce a mathematical framework for computing 
these metrics and then determining whether differences in 
these metrics are statistically significant. For the benefit of 
the sustainability community, we have provided open source 
software that implements our comparison methodology in 
the MATLAB programming language. A case study of ex- 
periments on our building-wide testbed shows the utility of 
the proposed techniques. 

2. EXISTING COMPARISON METHODS 

The International Performance Measurement and Verifica- 
tion Protocol (IPMVP) [6] classifies approaches to tracking 
energy usage into four general classes, and it is summarized 
in Table [I] Though IPMVP is a standard protocol for eval- 
uating energy usage of different building components, here 
we restrict our discussion to HVAC equipment. One class in 
IPMPV is called Option A and refers to measuring key vari- 
ables that affect HVAC energy consumption. An example 
is measuring the OAT and the SAT settings. Another class 
is called Option B and involves directly measuring energy 
usage of the HVAC. Option C is the situation in which the 
whole building or sub-building energy usage is measured, 
while Option D involves utilizing simulation models that es- 
timate HVAC energy consumption. 

The approach in [3], where the energy consumption of two 
control schemes on a single-room testbed with central air 
conditioning were compared, was a combination of Option 
B and Option D. Mathematical models of the temperature 
dynamics of different control schemes and their energy char- 
acteristics were constructed in order to allow comparisons 
of experimentally measured HVAC energy usage to simula- 
tions over identical weather and occupancy conditions. Un- 
fortunately, this method does not easily scale from the small 
system considered in [3j to building-sized HVAC systems be- 
cause of additional complexities of building-sized systems. 

In [l8] , the energy usage of two controllers for a ceiling ra- 
diant heating system was compared using an Option A ap- 
proach. The temperature difference between the OAT and 
the set point for a loop of hot water was used to evaluate 
the energy efficiency of different control methods. The gen- 
eral challenge with using Option A is that, depending on the 
particular HVAC equipment, the energy usage can scale in 



complex ways as a function of the key parameters that are 
measured. Moreover, auxiliary equipment in an HVAC sys- 
tem (e.g., pumps, fans) often significantly contribute to en- 
ergy usage; such parameters and their relationship to energy 
consumption are not usually measured or well understood. 

Optimization of thermal storage for campus-wide building 
cooling was considered in [m], and the energy usage of dif- 
ferent controllers was compared using an Option B approach 
coupled with a regression model of baseline performance: 
This indicates large energy savings when using novel con- 
trollers. Such an analysis is only possible when direct en- 
ergy measurements of the equipment are available; when this 
is not the case, the differences in measurements of two con- 
trollers can often be a small percentage of the total building- 
wide energy consumption. A regression model alone cannot 
determine whether such differences are statistically signifi- 
cant. 

We believe that a more unbiased approach for comparing 
HVAC is Option C combined with simple models. Note 
that our proposed method also applies to Option B, when 
the HVAC energy usage is directly measured. Experiments 
on our testbeds indicate a significant impact of OAT on 
the energy characteristics of HVAC systems. Our proposed 
method is to use nonparametric modeling methods to com- 
pute quantitative metrics of energy usage and occupant com- 
fort, as they relate the OAT. This framework allows for addi- 
tional nonparametric tools that can determine whether the 
differences in energy usage and occupant comfort of different 
HVAC controllers are statistically significant. 

3. EXPERIMENTAL SETUP 

In order to ensure fair comparisons between different con- 
trol methods, it is imperative that the experiments be con- 
ducted using identical building and HVAC configurations. 
For the sake of argument, suppose that controller 1 is the 
manufacturer's configuration, and controller 2 is identical to 
controller 1 except for that it turns the HVAC off during 
the night time. Then controller 2 can achieve substantial 
energy savings, but this does not reflect savings due to con- 
trol schemes: The energy savings are due to having different 
configurations of comfort levels. 

When comparing HVAC controllers, settings related to occu- 
pant comfort should be kept constant because the majority 
of energy is used in maintaining comfort. The specific set- 
tings that need to be kept equal will vary depending upon 
the building, but important variables to consider include 
desired temperatures in different building zones, allowable 
amounts of temperature deviation in these zones, and min- 
imum and maximum amounts of air flow. 

There is another experimental issue that is subtle: It is in- 
correct to keep repeating the analysis as more experimen- 
tal data is measured, without making corrections to the 
methodology. The reason is that the probability of making 
errors accumulates each time the hypothesis testing method- 
ology is used with additional data. Corrections would need 
to be made to ensure that the accumulated probability of 
error does not grow too large. This can be done in a prin- 
cipled manner using techniques from sequential hypothesis 
testing (e.g., [20] [15] [8]); though, we do not discuss in our 
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Figure 1: The scatter plot shows the data points 
(To[i], -E[i]), and the smooth curve is the estimated en- 
ergy characteristic E ,x(To) for an HVAC controller. 



paper on how to do so. Here, we assume that the analysis 
is conducted once with a fixed amount of data. 



4. MEASURING ENERGY EFFICIENCY 

Without loss of generality, we assume that building-wide 
measurements of energy usage are available at hourly inter- 
vals. These values will be denoted as E[i] for i — 1, . . . ,N 
measurements. Again, note that this framework also ap- 
plies to the situations where energy usage is measured at 
daily intervals and where the HVAC energy usage is directly 
measured. Furthermore, we assume the availability of OAT 
measurements that correspond to the energy usage measure- 
ments: T a [i] for i — 1, . . . , TV. 

4.1 Quantifying Energy Consumption 

The general model describing the relationship between en- 
ergy usage, OAT, occupancy O, and other factors (e.g., solar 
effects, equipment, etc.) X is 



E = f(T o ,0,X), 



(1) 



where /(•,-,•) is a nonlinear relationship that is unknown. 
However, occupancy and other factors are generally not di- 
rectly measured (though their effects on the thermal dynam- 
ics and energy usage can be estimated using semiparametric 
regression [3] [5]). As a result, it is typically only possible 
to consider the energy usage averaged over occupancy and 
other factors 



E , X (T ) =E[f{T o ,0,X)\T o ]. 



(2) 



Equation pj can be estimated using nonparametric regres- 
sion M. (We use local linear regression in the provided 
code.) It is not a single value, but is rather a curve that 




Figure 2: The solid line indicates the set point of a 
zone throughout a single day, and the extent of the 
dashed lines describes the band of comfort. 



describes the relationship between the OAT and the av- 
erage energy consumption. Intuitively, if the data points 
(T [i], E[i]) for i = 1, . . . ,N represent a scatter plot of en- 
ergy usage versus OAT; then the estimated curve E 0tX (T ) 
represents the smoothed version of the scatter plot. An il- 
lustration of this can be seen in Fig. [T] 

The average amount of energy used in one hour is therefore 

E g =K g (E 0rX (T )) =K g [E[f(To,0,X)\T o ]], (3) 

where g(T ) is a probability distribution of OATs. This no- 
tation allows us to define the average amount of energy used 
in one day as Ed ay = ^9»> where gi{T ) is the proba- 

bility distribution of OAT during the i-th hour of the day. 
For simplicity of calculations, the source code we provide 
assumes a uniform distribution for the OAT. 



4.2 Interpretation of Energy Characteristics 

It is well known (e.g., [II]) that the energy characteristics 
of HVAC have the following qualitative features: Energy us- 
age is lowest at moderate OATs, and the consumption of 
energy increases as the OAT increases or decreases. Physi- 
cally, the minimum in the energy characteristic corresponds 
to the OAT at which the building switches between pre- 
dominantly heating and predominantly cooling. This can 
be seen in Fig. [TJ which uses measurements taken from our 
building-wide HVAC testbed. The quantitiative features of 
the energy characteristics vary depending upon the HVAC 
controller and other specific characteristics of the building 
and its weather microclimate. 

4.3 Comparing Energy Consumption 

We make the assumption that exactly two different control 
schemes are being compared; such an assumption is not re- 
strictive, because we can do a set of pairwise comparisons 
with appropriate corrections made for multiple hypothesis 
testing. The two controllers being compared will be re- 
ferred to as controller 1 and controller 2, and they will be 
denoted using a superscript 1 and 2, respectively. Though 
we could compute the average amount of energy used in 
one day for each controller, there is no guarantee a priori 
that the difference of these estimated energy characteristics 
Eo, x {T ), Eo X (T ) is statistically significant 
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Figure 3: The scatter plot shows the data points 
(To[i], C[i]), and the smooth curve is the estimated 
comfort characteristic C' 0tX (To) for an HVAC con- 
troller. 



Our approach is to use a hypothesis test to quantify the 
evidence for the statement that the estimated characteris- 
tics for the two controllers are equal: E 1 tX {T ) = E 2 X (T ). 
If the p- value of this test is less than a significance level a, 
then we can say that the difference is statistically significant. 
Otherwise, the difference is not statistically significant if the 
p- value is greater than a. The code we provide uses a sig- 
nificance level of a = 0.01, and note that the intuition is 
that a gives the probability of incorrectly concluding that a 
difference is statistically significant. 

Because the curves E 1 X {T ),E 2 X {T ) are computed using 
nonparametric methods, it is not possible to use common 
methods like the t-test to compare them. Nonparametric 
hypothesis tests based on bootstrap procedures are an at- 
tractive alternative for our situation 19 . However, there is 



a temporal correlation between the measured energy usage 
E[i] for different values of i. This precludes the use of re- 
sampling residuals bootstrap, which is a standard bootstrap 
method. 

The methodology we propose makes use of a nonparamet- 
ric hypothesis test that utilizes the moving block bootstrap. 
This form of bootstrap is designed to handle dependent data 
in which there is a temporal correlation between measure- 
ments. The interested reader can refer to [l9j for details 
about this hypothesis test, and this is the technique that is 
implemented in our code. This is used to determine whether 
El x (T ), E 2 x (T a ) are statistically different. We can also 
use the same methodology to determine whether the differ- 
ence in the estimated average energy usage over one day 
AE 2 ' 1 — E\ ay — E\ ay is not equal to zero to a statisti- 
cally significant level. Note that because two hypotheses are 
being tested, we must make appropriate adjustments [151: 




Figure 4: Sutardja Dai Hall is 141, 000-square- foot 
building, and our BRITE-S testbed is within this 
building. 



Our code uses a Bonferroni correction to generate adjusted 
p- values |21| . 

4.4 Confidence Intervals of Energy Consump- 
tion 

If the hypothesis test in Sect. 4.3 indicates that AE 2 ' 1 7^ 
to a statistically significant level, then we can interpret both 
the value and magnitude of this quantity. If AE 2 ' 1 < 0, then 
this means that controller 2 uses less energy than controller 
1 on average, and the absolute value \AE • \ is the average 
amount of energy savings over a day due to controller 2 as 
compared to controller 1. The opposite statement holds if 
AE 2 - 1 > 0. 

Since AiJ 2 ' 1 is an estimated quantity, it will itself have un- 
certainty. In order to better characterize the difference be- 
tween the energy consumption of controllers 1 and 2, it is 
useful to also compute a confidence interval for AE 2 ' 1 . For 
a confidence level of /3 £ (0, 1], the confidence interval con- 
tains the true quantity (e.g., AE 2 ' 1 ) for /3 fraction of the ex- 
periments. The same moving block bootstrap methodology 
used in Sect. 14.31 can be used to estimated a bias-corrected 
bootstrap confidence interval [7J. 

5. MEASURING OCCUPANT COMFORT 

Quantifying the comfort levels of occupants is difficult be- 
cause it is a function of many variables: metabolic rate, 
clothing insulation, air temperature, radiant temperature, 
air speed, and humidity [I] . This problem is further compli- 
cated by the fact that most buildings are not instrumented 
to measure these variables. 

In order to define a quantification of comfort that is both 
tractable and will scale to many buildings, we focus on a 
measure that is only dependent on temperature; temper- 
ature measurements are usually available through existing 
thermostats. We assume that the average temperature for 
each of the j = 1, . . . , Z zones are measured at hourly inter- 
vals Tj[i] for i = 1, ...,N. 



5.1 Band of Comfort 



Controller 2 


Energy Characteristics 


AE 2 ' 1 


Comfort Characteristics 


AC 2 ' 1 


Temperature and 
Reheat Schedule 


Statistically Different 
(p = 0.002) 


-1.16MWh 
(p = 0.08) 


Statistically Different 
(p = 0.002) 


0.35°F 
(p = 0.9) 


LBMPC SAT 
Control 


Statistically Different 
(p = 0.002) 


-1.53MWh 
(p = 0.002) 


NOT Statistically Different 
(p = 0.8) 


-0.75°F 
(p = 0.5) 



Table 2: Summary of case studies on the BRITE-S testbed. Controller 1 is the default, manufacturer- 
provided controller, and controller 2 is as indicated. Also, ot = 0.05 was taken to be the significance level of 
the hypothesis tests. 



The jr'-th zone of a building will have a temperature set point 
Sj that can vary throughout the day, and we assume that we 
have recorded the average value of this set point at hourly in- 
tervals: Sj[i] for i — 1, . . . ,N. Furthermore, we assume that 
comfort is defined as maintaining zone temperature within 
±£>j [i] for i = 1, . . . , N of the set point. It can be visual- 
ized as a band of comfort, as shown in Fig. [2] This band is 
allowed to vary because the configured comfort levels often 
depend on the time of day or week; an example is allow- 
ing greater temperature fluctuations in an office building at 
night. 

Our justification for this notion of comfort is that it corre- 
sponds to the Class A, Class B, and Class C notions of com- 
fort as defined in |T[. Furthermore, this quantification is im- 
plicitly making the assumption that the set point indicates 
the true occupant preference. It represents a compromise 
between tractability of quantifying comfort and maintain- 
ing a reasonable metric. 

5.2 Quantifying Occupant Comfort 

Let (x)+ be the thresholding function, which is defined so 
that (x) + = if x < and (x)+ — x otherwise. We define 
our quantification of comfort using soft thresholding as 



C = 1/Z • Ef =1 /o(m - Si I - Bj) + dt, 



(4) 



where the integral with respect to dt is over one hour of time. 
The intuition is that this quantity increases whenever a zone 
temperature leaves the band of comfort, and the amount of 
increase in this quantity is proportional to the amount and 
duration of temperature deviation. 



where g(T ) is a probability distribution of OATs. We can 
then define the average amount of comfort in one day as 
Cday = Siii Css> where gi(T ) is the probability distribu- 
tion of OAT during the i-th hour of the day. Our source 
code assumes a uniform distribution for the OAT. 

5.3 Interpretation of Comfort Characteristics 

The purpose of an HVAC system is to provide uniform levels 
of occupant comfort regardless of external and internal con- 
ditions, and so the ideal HVAC controller will provide a com- 
fort characteristic that is constant across all OATs. How- 
ever, in practice it is more difficult for the HVAC to maintain 
the building environment when the weather is more extreme. 
This leads to slight decreases in comfort (i.e., the comfort 
characteristic is higher) at low and high OATs. This can 
be seen in Fig. |3j which uses measurements taken from our 
building-wide HVAC testbed. The quantitative features of 
the comfort characteristics vary depending upon the HVAC 
controller and other specific characteristics of the building 
and its weather microclimate. 

5.4 Comparing Occupant Comfort 

Using the hypothesis testing methodology outlined in Sect. 
|4.3| we can determine (a) whether the estimated comfort 
characteristics Cl X (T ),C 2 X (T ) for the two controllers are 
statistically different, and (b) whether the difference in the 
estimated average comfort over one day AC 2,1 = C 2 ay — 
C\ ay is not equal to zero to a statistically significant level. 
Like in Sect. |4.3[ we must correct for multiple comparison 
effects. 



Because Tj is implicitly a function of the outside temper- 
ature To, occupancy O, and other factors X, we can ab- 
stractly represent the comfort quantification as 



C = h{T ,O,X), 



(5) 



where h(-,-,-) is a nonlinear relationship that is unknown. 
Just as in the case of energy, we consider the comfort aver- 
aged over occupancy and other factors 



C O)X (T ) =E[h(T ,O,X)\T ], 



(6) 



because they are not measured. We can again generate a 
scatter plot of the data points (T [i], C[i]) for i — 1, . . . , N, 
where C[i] — X^d^iH — — Bj[i]) + . And a smoothed 

version can be estimated C 0>x (To) using nonparametric re- 
gression. An example of this is seen in Fig. [3] 

The average occupant comfort over one hour is given by 

C g = E g (C o , x (T )) =E g [E[h{T ,O,X)\T o ]], (7) 



5.5 Confidence Intervals for Occupant Com- 
fort 

indicates that AC 2,1 =fi 



4.3 



If the hypothesis test in Sect, 
to a statistically significant level, then we can interpret both 
the value and magnitude of this quantity. Because lower 
values of C correspond to greater comfort levels, we can 
interpret the quantit y AC * 2 ' 1 in the same way as AE 2 ' 1 is 



interpreted in Sect. 



4.4 



And because AC 



is an esti- 
mated quantity, the difference between the comfort levels 
of controllers 1 and 2 can be better characterized by also 
computing its confidence interval as in Sect. |4.4| 

6. CASE STUDY: BRITE-S TESTBED 

The Berkeley Retrofitted and Inexpensive HVAC Testbed 
for Energy Efficiency in Sutardja Dai Hall (BRITE-S) plat- 
form [12] [| is a building-wide HVAC system that main- 
tains the indoor environment of a 141,000-square-foot build- 
ing, shown in Fig. |4j that is divided between a four-floor 
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Figure 5: The cross marks and solid line (points 
and dashed line) denote the energy characteristics 
of the schedule (default) controller. 



Figure 6: The cross marks and solid line (points 
and dashed line) denote the comfort characteristics 
of the schedule (default) controller. 



nanofabrication laboratory (NanoLab) and seven floors of 
general space (including office space, classrooms, and a cof- 
fee shop). The building automation equipment can be mea- 
sured and actuated through a BACnet protocol interface. 

The HVAC system uses a 650-ton chiller to cool water. Air- 
handler units (AHUs) with variable- frequency drive fans dis- 
tribute air cooled by the water to variable air volume (VAV) 
boxes throughout the building. Since the NanoLab must 
operate within tight tolerances, our control design can only 
modify the operation of the general space AHUs and VAV 
boxes, with no modification of chiller settings that are shared 
between the NanoLab and general space. 

The default, manufacturer-provided controller in BRITE- 
S uses PID loops to actuate the VAV boxes and keeps a 
constant SAT within the AHUs. Conventional SAT reset 
control is not possible because several VAV boxes for zones 
with computer equipment provide maximum air flow rates 
throughout the entire day for all, except the coldest SATs. 

Here, we present case studies that compare this default con- 
troller (controller 1) with two different controllers. Con- 
troller 2 will refer to the controller being compared to con- 
troller 1. Note that the building is configured to provide a 
band of comfort of ±Bj[i] — ±1°F for all times and zones. 
The results of these case studies are summarized in Table |2j 
and details are given below. 

6.1 Temperature and Reheat Schedule 

The first controller to be compared to controller 1 turns 
off all heating in the building during the night time, and 
it matches the SAT to the OAT in an effort to reduce the 
amount of energy used to cool the air. We call this the sched- 
ule controller. An Option A analysis that neglects auxiliary 



equipment like fans will estimate nearly zero energy usage. 
However, our comparison methodology shows that this is 
not correct. 

The estimated energy characteristics E„ >x (To), E 2 tX (To) are 
shown in Fig. |5j and their differences are statistically sig- 
nificant (p — 0.002). However, the estimated difference in 
average energy usage over one day AE 2 ' 1 = — f.l6MWh is 
not statistically significant (p = 0.08), meaning that there is 
not enough evidence to exclude that AE 2 ' 1 = OMWh. 

The estimated comfort characteristics C„ x (Tq), C 2 x (Tq) are 
shown in Fig. [6] and their differences are statistically sig- 
nificant (p — 0.002). However, the estimated difference in 
average comfort over one day AC* 2 ' 1 = 0.35°F is not statisti- 
cally significant (p = 0.9), meaning that there is not enough 
evidence to exclude that AC 2,1 = 0°F. 

Though the schedule controller saves energy by turning off 
heating at night, these savings are somewhat negated by 
having to heat the building to a comfortable temperature 
during the day (i.e., the hump in E 2 x (To) at 55°F). As 
compared to the default controller, the schedule controller 
substantially degrades comfort at colder temperatures and 
provides slight improvements in comfort at moderate tem- 
peratures. 

6.2 LBMPC Control of Supply Air Tempera- 
ture 

The second controller to be compared to controller 1 uses a 
hybrid system learning-based model predictive controller [5] 
[3] to determine a sequence of SATs amongst the values of 
52°F, 58°F, and 64°F. An Option A analysis provides overly 
optimistic estimates of the energy savings due to this con- 
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Figure 7: The cross marks and solid line (points and 
dashed line) denote the energy characteristics of the 
LBMPC (default) controller. 



Figure 8: The cross marks and solid line (points 
and dashed line) denote the comfort characteristics 
of the LBMPC (default) controller. 



trol method, while the methodology described in this paper 
shows modest energy savings. 

The estimated energy characteristics E„ tX (To), E„ iX (To) are 
shown in Fig. [7J and their differences are statisticaily signifi- 
cant (p = 0.002). The estimated difference in average energy 
usage over one day A.E 2 ' 1 — — 1.53MWh is statistically sig- 
nificant (p = 0.002). And the 95% confidence interval is 
AE 2 ' 1 6 [-2.07,-1.02]MWh. 

The estimated comfort characteristics Cl }X (To), C% tX (To) are 
shown in Fig. [8] and their differences are not statistically 
significant (p = 0.8). Furthermore, the estimated difference 
in average comfort over one day AC* 2 ' 1 = -0.75°F is not 
statistically significant (p — 0.5), meaning that there is not 
enough evidence to exclude that AC 2,1 = 0°F. 

The LBMPC controller provides modest energy savings at 
most OATs, which sum up to significant savings over a day. 
And because the difference in comfort characteristics is not 
statistically significant, this suggests that the LBMPC and 
default controllers provide comparable levels of comfort. 

7. CONCLUSION 

We have presented quantitative metrics for comparing the 
energy usage and comfort of different HVAC controllers. 
These metrics can be computed relatively easily for a wide 
variety of buildings and are amenable to methods that can 
determine whether experimental differences between con- 
trollers are statistically significant. Though our focus here 
has been on energy-efficient HVAC control, this methodol- 
ogy can be used to compare the efficiency of different build- 
ing components such as water heating, lighting, and even 
changes in the comfort level of HVAC. 
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